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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address « 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 
Failure to reply within the set or~extehde~d period for reply will; by "statute", cause the application to become ABANDONED (35 U.S:C: § 1 33). - - 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )M Responsive to communication(s) filed on 24 March 2003 . 
2a)E3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) M Claim(s) 25-53 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) E3 Claim(s) 25-53 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

11) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)DAII b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2.D Certified copies of the priority documents have been received in Application No. . 



3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



14)0 



Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 
15)D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 

Attachment(s) 

1 ) (3 Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-41 3) Paper No(s). . 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) O Notice of Informal Patent Application (PTO-152) 

3) O Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) Q Other: 
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DETAILED ACTION 

1. This communicarion is in response to amendment filed 03/24/03, claims 1-24 have been 
cancelled, currently claims 25-53 remain pending, these are hereby set forth for examination. 

Claim Rejections - 35 USC § 103 

2. Quotation of 35 U.S.C. §103(a) which forms the basis for all obviousness rejections set forth in 
this Office action may be found in previous office action. 

3. Claims 25-53 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sezan et. al. (Sezan) 
MPEG-7 Standardization Activities in view of Bergman et. al. (Bergman) U.S. Patent No. 6,564,263 Bl. 

Regarding claim 25, Sezan teaches features of the invention substantially as claimed, teaching a 
system/method comprising: 

identifying multimedia categories (types) from received multimedia contents (introduction 
section, page 517); 

extracting multimedia objects to generate multimedia object descriptions from the multimedia 
content for a multimedia (type) (feature extraction, page 518, descriptors section page 518, Fig. 1, 
description generation); 

generating, from the multimedia object descriptions (description, page 518), an entity relationship 
representation (graph descriptions) for a multimedia type (features hierarchy section, page 5 19); and 

however prior art does not explicitly teach integrating the multimedia object descriptions and 
entity relation graph descriptions to generate a description record to represent content embedded within 
the multimedia content; 

aggregating, incorporating or combining, i.e. integrating from the multimedia object descriptions 
(col 12/lines 17-26), entity relation graph descriptions (col 3/lines 27-36, 46-51, 59-62, Figs. 3-7, col 
4/lines 20-30, integrated multimedia descriptions and entity relationships descriptions: col 6/lines 57-64, 
col 8/lines 42-67, entity-relationships, col 9/lines 55-col 10/line 10) to generate a composite object 
("description record"); 

wherein a composite object supports embedding of multimedia contents (col 7/lines 2-6, 
embedded hyper-linking, col 15/lines 4-17, spatial relationship of object within/contained, col 1 7/lines 
10-19, object within another object, col 1 7/lines 49-52); additionally teaching 
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identifying multimedia type or modalities in multimedia content (col 3/lines 21-36, description 
data type, col 12/lines 23-64, select identified type: col 19/lines 41-49); 

capture (extract) multimedia object features, attributes or modalities from multimedia objects to 
form multimedia object descriptions and entity relation descriptions from the multimedia object (col 
3/lines 37-51, cql_6/lines 39^ 

generating entity relationships descriptions and multimedia object descriptions from a multimedia 
category, type or modality (develop: col 14/lines 44-48, create: col 12/lines 43-50, generate: col 19/lines 
50-53); 

It would have been obvious to one ordinary skilled in the art at the time the invention was made 
to include Bergman's teachings for aggregating, incorporating or combining, i.e. integrating the 
multimedia object descriptions and their respective entity relation graph descriptions to generate a 
description record to represent content embedded within the multimedia content, motivation would be to 
generate entity relation description based on the multimedia object descriptions for multiple multimedia 
content types including a composite multimedia object description that represents content embedded 
within the multimedia content, as taught by Bergman. 

Regarding claim 26, multimedia object pyramid (hierarchy) descriptions for one of the multimedia types 
(Figs. 3-4 & 8, hierarchical model of multimedia object descriptions based on content type; col 8/lines 12- 
16, 20-67, Sezan: feature hierarchy, page 519). 

Regarding claim 27, the multimedia types includes image (Sezan: media type, page 519). 

Regarding claim 28, separating (segmenting) each multimedia content into descriptor defining portions 
(segments) including content from one of the multimedia type for the multimedia content (Sezan; page 
518); and 

generating one feature representing a feature for one of the portions by feature extraction and 
annotation (Bergman: feature description: col 8/lines 42-46, annotation: col 8/lines 55-58, extraction col 
3/lines 37-51); 

wherein the generated multimedia object descriptions comprises one feature description for one 
segment (Bergman; col 8/lines 20-41). 

Regarding claim 29, wherein the segments are selected from the group consisting of local segments and 
global segments (Bergman: col 8/lines 55-58). 





Application/Control Number: 09/495,175 (HUANG et. al.) 
Ai1 Unit: 2142 



Page 3 



Regarding claim 30, feature description from the group consisting of media, semantic and temporal 
features (Bergman: col 8/lines 42-67). 

Regarding claim 31, a feature description selected from the group consisting of data location, scalable 
repres_entation and modality transcoding (Bergman: col 9/lines 25-37). 

Regarding claim 32, wherein the semantic features are further defined by one feature description selected 
from the group consisting who (Bergman: col 8/lines 47-50). 

Regarding claim 33, temporal features are further defined by one feature description consisting of 
duration (Bergman: objects duration, col 16/lines 16-34). 

Regarding claim 34, 

generating media object descriptions from the multimedia segment for one of the multimedia 
types by media object extraction processing (Sezan: feature extraction, page 518, descriptors section page 
518, Fig. 1, description generation, Bergman: capture (extract) multimedia object features, col 3/lines 37- 



generating media object hierarchy descriptions from the generated media object descriptions by 
object hierarchy construction and extraction processing (Bergman: col 14/lines 44-48; Sezan: description, 
page 518, features hierarchy section, page 519); and 

generating media entity relation graph descriptions from the generated media object descriptions 
by entity relation graph generation processing (Bergman: col 3/lines 27-36, 46-51, 59-62, Figs. 3-7, col 
4/lines 20-30, col 6/lines 57-64, col 8/lines 42-67, entity-relationships, col 9/lines 55-col 10/line 10). 

Regarding claim 35, segmenting the content of each multimedia type in the multimedia object into 
segments within the multimedia object by media segmentation processing (Bergman: col 3/lines 21-36, 
col 12/lines 23-64; Sezan: categories (types) introduction section, page 517); 

generating one feature description for one of the segments by feature extraction and annotation 
(Bergman: feature description: col 8/lines 42-46, annotation: col 8/lines 55-58, extraction col 3/lines 37- 



wherein the generated media object descriptions comprise the feature description for one of the 
segments (Bergman; col 8/lines 20-41). 



51); 



51); 
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Regarding claim 36, substantially the same as claim 30, same rationale of rejection is applicable. 

Regarding claims 37-40, wherein generating media object pyramid (hierarchy) descriptions generates 
terminal/composite objects define as multimedia content pyramid (hierarchy) descriptions of the media 
object descriptions (Bergman: col 3/lines 27-36) based on relationships of media objects represented by 
the media object descriptions (Bergman: col 3/lines 41-51), and wherein the relationships consisting of 
media feature relationships, semantic feature relationships, temporal feature relationships, and spatial 
feature relationships (Bergman: col 6/line 1-14, 57-67, col 7/lines 10-14, 20-25). 

Regarding claims 4 1 and 46, wherein generating media entity relation graph descriptions (Bergman: col 
9/lines 55-col 10/line 10) generates entity relations graph descriptions of the media object descriptions 
based on entity-relationships and dependency-entity relationships of media objects represented by the 
media object descriptions (Bergman: col 14/lines 45-48, col 1 9/lines 50-53), 

wherein the relationships are selected from the group consisting of media feature relationships, 
semantic feature relationships, temporal feature relationships and spatial feature relationships (Bergman: 
col 6/lines 1-14, 57-67, col 7/lines 10-14, 20-25). 

Regarding claims 42-45, wherein generating multimedia object pyramid (hierarchy) descriptions 
generates multimedia object hierarchy descriptions of the multimedia object descriptions based on media 
content relationships of multimedia terminal/composite objects represented by the multimedia object 
descriptions (Bergman: col 11/line 27-33, inter-feature relationships: col 19/lines 50-53, inter-object 
relationships, col 15/lines 10-15); based on temporal, spatial feature relationships of multimedia objects 
(Bergman: col 15/lines 10-15); and semantic feature relationships of multimedia object (Bergman: col 
6/lines 58-64). 

Regarding claim 47, receiving (Sezan: page 517) and transcoding (encoding) the multimedia object 
descriptions into encoded description information (Bergman: col 7/lines 20-24), and storing the encoded 
description information as one (description) record (Sezan: left column, page 518). 

Regarding claim 48, this claim combines limitation(s) substantially the same as claims 25, and 47, same 
rationale of rejection is applicable 

Regarding claim 49 & 50, the encoding comprises binary encoding (Bergman: col 1 3/lines 3 1-33). 
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Regarding claims 51 & 52, the encoding comprises the extensible Markup Language (XML) encoding 
(Bergman: col 14/lines 4-18). 

Regarding claim 53, includes limitations substantially the same as claim 25, same rationale of rejection is 
applicable^ and further 

generating, from the multimedia object descriptions, multimedia object pyramid (hierarchy) 
descriptions (Bergman: generating entity relationships descriptions and multimedia object descriptions 
from a multimedia category, type or modality develop: col 14/lines 44-48, create: col 12/lines 43-50, 
generate: col 19/lines 50-53) by object hierarchy construction and extraction processing, for a 
multimedia content modality or feature (type) (extract: col 3/lines 41-51, col 6/lines 39-42, construct: col 
12/lines 43-50); and 

integrating the multimedia object descriptions and the multimedia object pyramid (hierarchy) 
descriptions to generate a descriptor (description record) to represent content embedded within the 
multimedia content (Bergman: integration, col 12/lines 17-26, entity relation graph descriptions, col 3/lines 27-36, 
46-51, 59-62, Figs. 3-7, col 4/lines 20-30, integrated multimedia descriptions and entity relationships 
descriptions: col 6/lines 57-64, col 8/lines 42-67, entity -relationships, col 9/lines 55-col 10/line 10) to 
generate a composite object); 

wherein a composite object supports embedding of multimedia contents (col 7/lines 2-6, 
embedded hyper-linking, col 15/lines 4-17, spatial relationship of object within/contained, col 1 7/lines 
10-19, object within another object, col 1 7/lines 49-52). 



4. Applicant traverses previous rejection on the basis that the prior does not teach new claim 
limitations specifically, integrating the multimedia object descriptions and the entity graph descriptions 
to generate a "description record" (information, data, object or descriptor) to represent "content 
embedded 55 within the multimedia content. 



Response to argument 



5. Applicants arguments with respect to claim 25-53 have been considered but are moot in view of 
the new ground(s) of rejection. 
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6. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office 
action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is 
reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for 
reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the 
event a firstreply is filed within TWO MONTHS of the mailing date of this final action and the advisory 
action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any extension fee 
pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this final 
action. 

Prosecution of this application is closed by means of this final office action § 1.1 13, applicant 
may request continued examination of the application by filing a Request for Continued Examination of 
under 37 CFR § 1.114 and providing the corresponding fee set forth in § IT 7(e) for the submission of, 
but not limited to, new arguments, an information disclosure statement, an amendment to the written 
description, claims, drawings, or new evidence in support of patentability. Or applicant whose claims has 
been twice rejected, may appeal from the decision of the administrative patent judge to the Board of 
Patent Appeals and Interferences under 35 U.S.C. §134. 

Any inquiry concerning this communication or earlier communications from the examiner should 
be directed to Prieto, B. whose telephone number is (703) 305-0750. The Examiner can normally be 
reached on Monday-Friday from 6:00 to 3:30 p.m. If attempts to reach the examiner by telephone are 
unsuccessful, the Examiner's Supervisor, Mark R. Powell can be reached on (703) 305-9703. The fax 
phone number for the organization where this application or proceeding is assigned is (703) 308-6606. 
Any inquiry of a general nature or relating to the status of this application or proceeding should be 
directed to the receptionist whose telephone number is (703) 305-3800/4700. 

Any response to this final action should be mailed to: 
Box AF 

Commissioner of Patents and Trademarks 
Washington, D.C. 20231 
or Faxed to: 

(703) 746-7238 for TC 2100 Official After-final communications; please mark 
"EXPEDITED PROCEDURE", and 

(703) 746-7239 for other TC 2 100 Official communications. 
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or: 

(703) 465-7240 for Non-Official, Draft communications, status query, please 
label "PROPOSED" or "DRAFT". 



or Telephone: 

(703) 306-5631 for TC 2100 Customer Service Office 

Hand-delivered responses should be brought to Crystal Park II, 2 12 1 Crystal Drive, Arlington. 
VA., Sixth Floor (Receptionist). 




TC2100 
Patent Examiner 



